Prompt Generator for Stable Diffusion#

Note

Install ekorpkit package first.

Set logging level to Warning, if you don’t want to see verbose logging.

If you run this notebook in Colab, set Hardware accelerator to GPU.

!pip install ekorpkit[art] exit()

Preparing the environment#

%config InlineBackend.figure_format='retina'
%load_ext autotime

from ekorpkit import eKonf

eKonf.setLogger("INFO")
eKonf.set_cuda(device=0)
print("version:", eKonf.__version__)

is_colab = eKonf.is_colab()
print("is colab?", is_colab)
if is_colab:
    eKonf.mount_google_drive()
project_dir = eKonf.set_workspace(workspace="/content/drive/MyDrive/workspace/", project="ekorpkit-book")
print("project_dir:", project_dir)

INFO:ekorpkit.base:Setting cuda device to ['A100-SXM4-40GB (id:0)']
INFO:ekorpkit.utils.notebook:Google Colab not detected.
INFO:ekorpkit.base:Setting EKORPKIT_WORKSPACE_ROOT to /content/drive/MyDrive/workspace/
INFO:ekorpkit.base:Setting EKORPKIT_PROJECT to ekorpkit-book
INFO:ekorpkit.base:Loaded .env from /workspace/projects/ekorpkit-book/config/.env

version: 0.1.40.post0.dev5
is colab? False
project_dir: /content/drive/MyDrive/workspace/projects/ekorpkit-book
time: 2.16 s (started: 2022-11-04 03:30:54 +00:00)

Load the Generator and Generate Prompts#

from ekorpkit.models.art.prompt import PromptGenerator

pgen = PromptGenerator(verbose=False)

2022-11-04 03:30:57.641122: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
INFO:ekorpkit.batch:Batch name: prompt-batch, Batch num: 1

time: 4.23 s (started: 2022-11-04 03:30:56 +00:00)

prompts = pgen.generate_prompts(prompt="examining documents")
prompts

INFO:ekorpkit.batch:Using existing path: /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen
INFO:ekorpkit.batch:Batch name: prompt-batch, Batch num: 2
INFO:ekorpkit.batch:Merging config with args: {}
INFO:ekorpkit.batch:Setting seed to 1772383741
INFO:ekorpkit.batch:Saving config to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-prompt/prompt-batch/configs/prompt-batch(2)_config.yaml

['examining documents, artstation trends beautiful digital anime painting of a happy cyborg child surrounded by bright cute cute little bubbles bursting out of the clouds with glowing lights, dramatic lighting, intricate, highly detailed, unreal engine, Trend',
 'examining documents. There is also a detailed matte painting of a village in the middle. art station, concept art, matte, sharp focus, illustration.. by michael welan, raymond swanland, and greg rut',
 'examining documents in an enchanted room surrounded by colorful glowing orbs, dark and ominous mood, volumetric lights, red and cyan theme, cinematic lighting, high quality, highly detailed, octane render, 8 k resolution, trending on artstation',
 'examining documents. holographic. photorealistic. highly detailed. cinematic. 4 k closeup portrait shot of beautiful pale gothic girl in white dress and necklaces surrounded by waterfalls with glowing butterflies, intricate,',
 'examining documents and videos, trending on artstation, 8k, HD portrait of danny devito smoking a cigar, intricate, elegant, glowing lights, highly detailed, digital painting, artstation, glamor pose,',
 'examining documents. intricate elegant. Trending on artstation. 8k a hyper detailed matte painting of a city landscape inside of a gigantic blackberry bushes. at night. dense foliage. aerial photography. extreme bokeh.',
 'examining documents and schematics, highly detailed, octane render, 8 k ultra realistic illustration of beautiful female japanese dragon princess, flowing hair, cyberpunk, intricate elegant ornate details, by alphonse',
 'examining documents, hyper detailed, ultra realistic, concept art, intricate details, eerie, highly detailed, photorealistic, octane render, 8 k, unreal engine. art by artgerm and greg rutkowski and charlie',
 'examining documents and detailed white background, elegant detailed digital painting, concept art, matte, sharp focus, 4k ultra hd, illustration, magical, unreal engine, octane render, cinematic lighting, wide angle establishing shot, extremely high detail',
 'examining documents and document insert an emoji with or without spaces, in a cyberpunk city, by Guy Denning, Johannes Itten, Derek Gores, Russ Mills, glitch art, octane render, iridescent accents']

time: 449 ms (started: 2022-11-04 03:52:24 +00:00)

Train a Generator#

pgen.train(model_name="ekorpkit/stable-prompts-2")

WARNING:datasets.builder:Using custom data configuration Gustavosta--Stable-Diffusion-Prompts-d22aeec0ba2a9fdb
WARNING:datasets.builder:Reusing dataset parquet (/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/cache/Gustavosta___parquet/Gustavosta--Stable-Diffusion-Prompts-d22aeec0ba2a9fdb/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)

Token indices sequence length is longer than the specified maximum sequence length for this model (2613888 > 1024). Running this sequence through the model will result in indexing errors
***** Running training *****
  Num examples = 18847
  Num Epochs = 1
  Instantaneous batch size per device = 1
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 2356
Automatic Weights & Biases logging enabled, to disable set os.environ["WANDB_DISABLED"] = "true"
ERROR:wandb.jupyter:Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
wandb: Currently logged in as: entelecheia. Use `wandb login --relogin` to force relogin
wandb version 0.13.4 is available! To upgrade, please run: $ pip install wandb --upgrade
Tracking run with wandb version 0.13.3
Run data is saved locally in /workspace/projects/ekorpkit-book/ekorpkit-book/docs/lectures/aiart/wandb/run-20221102_081028-3heixjgo
Syncing run ekorpkit/stable-prompts-2 to Weights & Biases (docs)
[2356/2356 04:59, Epoch 1/1]
Step Training Loss
200 3.776300
400 2.705100
600 2.517600
800 2.376600
1000 2.295400
1200 2.231800
1400 2.175900
1600 2.143700
1800 2.104900
2000 2.082700
2200 2.081700 

Training completed. Do not forget to share your model on huggingface.co/models =)
Waiting for W&B process to finish... (success).

Run history:


train/epoch▁▂▂▃▄▄▅▆▆▇▇█
train/global_step▁▂▂▃▄▄▅▆▆▇▇█
train/learning_rate█▇▇▆▅▄▄▃▂▂▁
train/loss█▄▃▂▂▂▁▁▁▁▁
train/total_flos▁
train/train_loss▁
train/train_runtime▁
train/train_samples_per_second▁
train/train_steps_per_second▁

Run summary:


train/epoch1.0
train/global_step2356
train/learning_rate0.0
train/loss2.0817
train/total_flos4924659858407424.0
train/train_loss2.38621
train/train_runtime353.5289
train/train_samples_per_second53.311
train/train_steps_per_second6.664

Synced ekorpkit/stable-prompts-2: https://wandb.ai/entelecheia/ekorpkit-book-prompt-generator/runs/3heixjgo
Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
Find logs at: ./wandb/run-20221102_081028-3heixjgo/logs
INFO:ekorpkit.models.art.prompt:Saving model to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2
Configuration saved in /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2/config.json
Model weights saved in /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2/pytorch_model.bin
INFO:ekorpkit.batch:Saving config to /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/outputs/stable-prompt/prompt-batch/prompt-batch(14)_config.yaml

time: 7min 9s (started: 2022-11-02 08:09:17 +00:00)

pgen.load_model(model_name="ekorpkit/stable-prompts-2")

INFO:ekorpkit.models.art.prompt:Loading model from /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2
loading configuration file /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2/config.json
Model config GPT2Config {
  "_name_or_path": "/content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2",
  "_num_labels": 1,
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_range": 0.02,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_inner": null,
  "n_layer": 6,
  "n_positions": 1024,
  "reorder_and_upcast_attn": false,
  "resid_pdrop": 0.1,
  "scale_attn_by_inverse_layer_idx": false,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 50
    }
  },
  "torch_dtype": "float32",
  "transformers_version": "4.21.2",
  "use_cache": true,
  "vocab_size": 50260
}

loading weights file /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2/pytorch_model.bin
All model checkpoint weights were used when initializing GPT2LMHeadModel.

All the weights of GPT2LMHeadModel were initialized from the model checkpoint at /content/drive/MyDrive/workspace/projects/ekorpkit-book/disco-imagen/models/ekorpkit/stable-prompts-2.
If your task is similar to the task the model of the checkpoint was trained on, you can already use GPT2LMHeadModel for predictions without further training.

time: 1.42 s (started: 2022-11-02 08:04:02 +00:00)

Generate images#

text_prompts = "a living tiger inside a gorgeous cave in a futuristic Jeju island, relaxing night, watercolour on the ceiling, glowing on glass, light art, dark, dark, gorgeous, dark, bright in an abandoned village in a futuristic alien nightscape"
batch_name = "cave-jeju"